`QCAlgorithm`'s `OptionChain()` api refactor #8334

jhonabreul · 2024-09-19T21:12:46Z

Description

This is a refactoring of the QCAlgorithm.OptionChain() API for fetching an option chain:

Changed return type from DataHistory<OptionUniverse> to OptionChain:
- Added DataFrame property to OptionChain to convert the option chain into a Pandas DataFrame.
Added the QCAlgorithm.OptionChains() method to fetch option chains for multiple symbols:
- Added DataFrame property to OptionChains to convert the option chain into a Pandas DataFrame. This dataframe will have a multi-index which first level is the canonical option. Indexing by the canonical symbol will result in a sub data frame equivalent to the one generated for a single OptionChain.

Additional changes:

Fix for list of base data convertion into Pandas DataFrame:
- This makes sure PandasConverter.GetDataFrame<T>(IEnumerable<T> data) doesn't assume the data is for a single symbol. For instance, if an option chain (IEnumerable<ContractData>) is to be converted to a data frame, each item will correspond to a single symbol, that is, each contract.
- This is fixed generically so that a single list can hold data for multiple symbols, regardless of the number of items for each symbol contained in the list.
Dataframes are now indexed by Symbol instances, instead of the symbol ID/Value string. This way, users can access it and use the Symbol directly for other operations, like manually adding option contracts.

Data frame before:

Sample:

                                                      open     high      low    close  volume  openinterest    value                                      id  impliedvolatility                                                                  greeks                                                                                                      csvheader     underlying
symbol                                time                                                                                                                                                                                                                                                                                                                                          
GOOCV WHEA9CWI9A86|GOOCV VP83T1ZUHROL 2015-12-24  462.000  466.700  456.950  462.000     0.0          23.0  462.000   GOOCV WHEA9CWI9A86|GOOCV VP83T1ZUHROL           1.050042  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  449.000  449.000  449.000  449.000     0.0          38.0  449.000   GOOCV W78ZEO2L4TK6|GOOCV VP83T1ZUHROL           0.000000  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  452.000  456.950  446.950  452.450     0.0           4.0  452.450   GOOCV WHEA9CS7IQ3Q|GOOCV VP83T1ZUHROL           1.341100  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  442.500  447.250  437.550  442.750     0.0           5.0  442.750   GOOCV WHEA9CWU5YBQ|GOOCV VP83T1ZUHROL           0.818821  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  429.800  434.450  424.500  430.050     0.0           2.0  430.050   GOOCV W78ZEO7DQDTY|GOOCV VP83T1ZUHROL           0.000000  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  430.500  434.800  425.000  430.650     0.0           0.0  430.650   GOOCV W8Z0KK90JWXY|GOOCV VP83T1ZUHROL           1.332343  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  433.000  437.750  427.950  433.000     0.0           2.0  433.000   GOOCV WHEA9CX04ADI|GOOCV VP83T1ZUHROL           0.830668  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  419.800  424.400  414.400  420.050     0.0          33.0  420.050   GOOCV W78ZEO7JOPVQ|GOOCV VP83T1ZUHROL           0.000000  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  418.700  418.700  418.700  418.700     0.0           1.0  418.700   GOOCV W8Z0KK96I8ZQ|GOOCV VP83T1ZUHROL           0.669938  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  423.000  427.900  417.950  423.450     0.0           0.0  423.450   GOOCV WHEA9CX62MFA|GOOCV VP83T1ZUHROL           1.031123  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54
                                      2015-12-24  409.900  414.500  404.450  410.050     0.0          19.0  410.050   GOOCV W78ZEO7PN1XI|GOOCV VP83T1ZUHROL           0.000000  QuantConnect.Data.UniverseSelection.OptionUniverse+PreCalculatedGreeks  symbol_id,symbol_value,open,high,low,close,volume,open_interest,implied_volatility,delta,gamma,vega,theta,rho  GOOG: ¤748.54

Data frame after (single option chain):

Sample:

                                        askprice  asksize  bidprice  bidsize     delta     expiry     gamma  impliedvolatility  lastprice  openinterest        rho  right  scaledstrike  strike  style  theoreticalprice     theta  underlyinglastprice underlyingsymbol      vega  volume
symbol

GOOCV WHEA9CZBGZ2E|GOOCV VP83T1ZUHROL   298.010        0   298.010        0  0.877012 2017-01-20  0.000153           1.702007    298.010          47.0   1.342803      0         460.0   460.0      0               0.0 -0.341970               748.54             GOOG  1.572933       1
GOOCV WRCOZE4RBLHI|GOOCV VP83T1ZUHROL   312.100        0   312.100        0  0.899879 2018-01-19  0.000478           0.335162    312.100           7.0   7.392764      0         460.0   460.0      0               0.0 -0.048927               748.54             GOOG  1.864050       4
GOOCV WRCOZE62Y9VQ|GOOCV VP83T1ZUHROL   247.000        0   247.000        0  0.909200 2018-01-19  0.000743           0.200133    247.000          32.0   9.385872      0         540.0   540.0      0               0.0 -0.033155               748.54             GOOG  1.731035       4
GOOCV WRCOZE6QRM2U|GOOCV VP83T1ZUHROL   221.880        0   221.880        0  0.937151 2018-01-19  0.000844           0.131673    221.880           9.0  10.757769      0         580.0   580.0      0               0.0 -0.023281               748.54             GOOG  1.294786       2
GOOCV W8Z0KK4PTCTI|GOOCV VP83T1ZUHROL   151.810        0   151.810        0  0.753193 2016-03-18  0.000513           1.699932    151.810         298.0   0.622079      0         600.0   600.0      0               0.0 -1.147463               748.54             GOOG  1.141951       1
GOOCV WHEA9CSPDQ92|GOOCV VP83T1ZUHROL   181.000        0   181.000        0  0.760117 2017-01-20  0.000801           0.497444    181.000         228.0   3.685862      0         600.0   600.0      0               0.0 -0.160227               748.54             GOOG  2.411593       3
GOOCV WBGM95B3K95Y|GOOCV VP83T1ZUHROL   129.900        0   129.900        0  0.820160 2016-06-17  0.001657           0.303375    129.900           5.0   2.316551      0         635.0   635.0      0               0.0 -0.128758               748.54             GOOG  1.363484       3
GOOCV WBGM92OO1A4M|GOOCV VP83T1ZUHROL   127.200        0   127.200        0  0.759528 2016-06-17  0.001422           0.419588    127.200           7.0   2.029567      0         640.0   640.0      0               0.0 -0.202744               748.54             GOOG  1.619014       1
GOOCV W6NBKMB4ITXI|GOOCV VP83T1ZUHROL    98.000        0    98.000        0  0.000000 2015-12-24  0.000000           0.000000     98.000          15.0   0.000000      0         650.0   650.0      0               0.0  0.000000               748.54             GOOG  0.000000      12
GOOCV W6U7P9VB241Y|GOOCV VP83T1ZUHROL    98.200        0    98.200        0  0.794604 2015-12-31  0.001908           1.434608     98.200          12.0   0.091437      0         650.0   650.0      0               0.0 -3.039695               748.54             GOOG  0.295543      12

Data frame after (multiple option chains):

Sample:

                                                                    askprice  asksize  bidprice  bidsize     delta     expiry         gamma  impliedvolatility  lastprice  openinterest       rho  right  scaledstrike  strike  style  theoreticalprice     theta  underlyinglastprice underlyingsymbol      vega  volume
canonical                   symbol                                                                                                                                                                                                                                                                                      
SPX 3JLXPT2SU|SPX 31        SPX WGFX5OWOQQ3Y|SPX 31                 1895.35        0   1895.35        0  0.999911 2016-12-16  2.000000e-07           0.926931    1895.35          41.0  0.972638      0         100.0   100.0      1               0.0 -0.003631              2038.97              SPX  0.007169       0
                            SPX WHEDSYPLF4Q6|SPX 31                 1892.55        0   1892.55        0  1.000000 2017-01-20  0.000000e+00           0.000000    1892.55        2500.0  1.070544      0         100.0   100.0      1               0.0 -0.002710              2038.97              SPX  0.000000       0
                            SPX WQEBVQ24LCJ2|SPX 31                 1856.15        0   1856.15        0  1.000000 2017-12-15  0.000000e+00           0.000000    1856.15          38.0  1.944604      0         100.0   100.0      1               0.0 -0.002686              2038.97              SPX  0.000000       0
                            SPX WGFX5OWUP25Q|SPX 31                 1796.40        0   1796.40        0  0.999829 2016-12-16  4.000000e-07           0.730009    1796.40           1.0  1.945658      0         200.0   200.0      1               0.0 -0.006759              2038.97              SPX  0.013243       0
                            SPX WHEDSYPRDGRY|SPX 31                 1793.75        0   1793.75        0  0.999021 2017-01-20  1.800000e-06           0.844378    1793.75        2500.0  2.109365      0         200.0   200.0      1               0.0 -0.012839              2038.97              SPX  0.069968       0
                            SPX WQEBVQ2AJOKU|SPX 31                 1758.55        0   1758.55        0  0.999738 2017-12-15  6.000000e-07           0.538417    1758.55           2.0  3.872808      0         200.0   200.0      1               0.0 -0.006395              2038.97              SPX  0.027962       0
                            SPX W792YA0AXWA6|SPX 31                 1738.80        0   1738.80        0  1.000000 2016-01-15  0.000000e+00           0.000000    1738.80          30.0  0.197131      0         300.0   300.0      1               0.0 -0.008214              2038.97              SPX  0.000000       6
                            SPX W87JLJT7MAWE|SPX 31                 1732.90        0   1732.90        0  0.999850 2016-02-19  5.000000e-07           1.441965    1732.90           0.0  0.478372      0         300.0   300.0      1               0.0 -0.014042              2038.97              SPX  0.004732       0
                            SPX W8Z4461XRFE6|SPX 31                 1729.05        0   1729.05        0  0.999788 2016-03-18  7.000000e-07           1.222505    1729.05          34.0  0.706966      0         300.0   300.0      1               0.0 -0.013817              2038.97              SPX  0.007956       0
                            SPX W9QOMSANWJVY|SPX 31                 1726.45        0   1726.45        0  0.999729 2016-04-15  8.000000e-07           1.087125    1726.45           0.0  0.934995      0         300.0   300.0      1               0.0 -0.013641              2038.97              SPX  0.011500       0
GOOCV 2U|GOOCV VP83T1ZUHROL GOOCV WHEA9CWI9A86|GOOCV VP83T1ZUHROL    462.00        0    462.00        0  0.921936 2017-01-20  1.765000e-04           1.050042     462.00          23.0  1.951418      0         290.0   290.0      0               0.0 -0.152740               748.54             GOOG  1.122288       0
                            GOOCV W78ZEO2L4TK6|GOOCV VP83T1ZUHROL    449.00        0    449.00        0  0.999931 2016-01-15  0.000000e+00           0.000000     449.00          38.0  0.180713      0         300.0   300.0      0               0.0 -0.005876               748.54             GOOG  0.000000       0
                            GOOCV WHEA9CS7IQ3Q|GOOCV VP83T1ZUHROL    452.45        0    452.45        0  0.912188 2017-01-20  1.513000e-04           1.341100     452.45           4.0  1.559289      0         300.0   300.0      0               0.0 -0.211365               748.54             GOOG  1.228025       0
                            GOOCV WHEA9CWU5YBQ|GOOCV VP83T1ZUHROL    442.75        0    442.75        0  0.928941 2017-01-20  2.104000e-04           0.818821     442.75           5.0  2.425643      0         310.0   310.0      0               0.0 -0.112672               748.54             GOOG  1.043158       0
                            GOOCV W78ZEO7DQDTY|GOOCV VP83T1ZUHROL    430.05        0    430.05        0  0.999931 2016-01-15  0.000000e+00           0.000000     430.05           2.0  0.192761      0         320.0   320.0      0               0.0 -0.006424               748.54             GOOG  0.000000       0
                            GOOCV W8Z0KK90JWXY|GOOCV VP83T1ZUHROL    430.65        0    430.65        0  0.950285 2016-03-18  2.120000e-04           1.332343     430.65           0.0  0.626794      0         320.0   320.0      0               0.0 -0.295237               748.54             GOOG  0.370133       0
                            GOOCV WHEA9CX04ADI|GOOCV VP83T1ZUHROL    433.00        0    433.00        0  0.922612 2017-01-20  2.217000e-04           0.830668     433.00           2.0  2.437955      0         320.0   320.0      0               0.0 -0.121860               748.54             GOOG  1.114780       0
                            GOOCV W78ZEO7JOPVQ|GOOCV VP83T1ZUHROL    420.05        0    420.05        0  0.999931 2016-01-15  0.000000e+00           0.000000     420.05          33.0  0.198784      0         330.0   330.0      0               0.0 -0.006698               748.54             GOOG  0.000000       0
                            GOOCV W8Z0KK96I8ZQ|GOOCV VP83T1ZUHROL    418.70        0    418.70        0  0.996359 2016-03-18  4.200000e-05           0.669938     418.70           1.0  0.760166      0         330.0   330.0      0               0.0 -0.021132               748.54             GOOG  0.036842       0
                            GOOCV WHEA9CX62MFA|GOOCV VP83T1ZUHROL    423.45        0    423.45        0  0.904063 2017-01-20  2.103000e-04           1.031123     423.45           0.0  2.093146      0         330.0   330.0      0               0.0 -0.175402               748.54             GOOG  1.312556       0

Related Issue

Closes #8331

Motivation and Context

Requires Documentation Change

How Has This Been Tested?

Unit tests
Regression algorithm

Types of changes

Bug fix (non-breaking change which fixes an issue)
Refactor (non-breaking change which improves implementation)
Performance (non-breaking change which improves performance. Please add associated performance test and results)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Non-functional change (xml comments/documentation/etc)

Checklist:

My code follows the code style of this project.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.
My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

Take into consideration data for multiple symbols in the same list

Also add pass-through properties for greek values in OptionUniverse

- Use OptionChains as output - Add DataFrame to OptionChain and OptionChains - Rename Greeks classes - Add ISymbolProvider for classes that have a symbol (IBaseData, OptionContract)

Also refactor OptionContract to handle: (1) Actual market data and option price model data, and (2) OptionUniverse data

…changes

Add NullGreeks class: keep ModeledGreeks as internal as possible

Address peer review

This allows propery hash and equality between Symbols, C# strings and Python strings

jhonabreul requested a review from Martin-Molinero September 19, 2024 21:12

jhonabreul force-pushed the bug-8331-option-chain-dataframe branch from dfffb99 to 8bd8117 Compare September 25, 2024 21:50

jhonabreul added 20 commits September 26, 2024 18:30

Fix pandas converter to handle list of data with different symbols

79369a5

Properly convert list of data into dataframe

11e3b9f

Take into consideration data for multiple symbols in the same list

Cleanup

be9d68c

Index dataframes by symbol object instead of SID string

862dc8d

Add symbol equality operator to compare against object

c2e3f7e

Exclude "ID" from option chain dataframe

8123c61

Minor fix

5138b1c

Add greeks columns directly in option chain dataframe.

b122ff4

Also add pass-through properties for greek values in OptionUniverse

Some cleanup

aa35111

Minor fix

3b649c3

Add new QCAlgorithm.OptionChains() method

407688d

- Use OptionChains as output - Add DataFrame to OptionChain and OptionChains - Rename Greeks classes - Add ISymbolProvider for classes that have a symbol (IBaseData, OptionContract)

Unify QCAlgorithmOptionChain API

b96a979

Also refactor OptionContract to handle: (1) Actual market data and option price model data, and (2) OptionUniverse data

Pass symbol properties to OptionUniverse option chain from algorithm

7b0db10

Format OptionContract for dataframe

1d632a9

Minor fix

6eb1df3

Add multiple option chains api regression algorithms and other minor …

80ef6a5

…changes

Address peer review

8839ea0

Add NullGreeks class: keep ModeledGreeks as internal as possible

Minor fix and add PandasConverter unit tests

52a9b89

Peer review: Non-thread-safe Lazy for Python

5ef2302

Handle Greeks unwrapping by PandasData

5d0e18d

jhonabreul force-pushed the bug-8331-option-chain-dataframe branch from fd88ae2 to 5d0e18d Compare September 26, 2024 23:17

jhonabreul added 3 commits September 27, 2024 10:42

PandasData cleanup

77b74f2

Add data and other minor changes

d58d78f

Unit test fix

3dbcc4b

jhonabreul changed the title ~~Dataframe from list of data with multiple symbols~~ QCAlgorithm's OptionChain() and OptionChains() api refactor Sep 27, 2024

jhonabreul changed the title ~~QCAlgorithm's OptionChain() and OptionChains() api refactor~~ QCAlgorithm's OptionChain() api refactor Sep 27, 2024

jhonabreul added 2 commits September 27, 2024 15:40

Update Pythonnet to 2.0.39

801b0f5

Cleanup

5b357b0

jhonabreul and others added 12 commits September 27, 2024 17:55

PandasData handling children class members

c8787d7

Address peer review

Fix: indexing symbol conversion in pandas mapper

f261e7b

Fix pandas mapper to convert string keys to symbol only when necessary

a27cea6

Cleanup

8c21d2a

Cleanup

fa129c0

Add PandasColumn python class to handle proper indexing

2574746

This allows propery hash and equality between Symbols, C# strings and Python strings

Minor fixes

39917a5

Symbol cache improvements

03d203f

Minor fix for cache miss

aa49a1b

Revert PandasMapper reserved names and improvements

6222d60

Minor fix

97c2793

Revert reserved names

d0f709c

Martin-Molinero approved these changes Oct 4, 2024

View reviewed changes

Minor fix for Symbol equality operators

3fd171a

jhonabreul closed this Oct 4, 2024

jhonabreul reopened this Oct 4, 2024

jhonabreul merged commit 0a9dc2c into QuantConnect:master Oct 4, 2024
10 of 14 checks passed

jhonabreul deleted the bug-8331-option-chain-dataframe branch October 4, 2024 16:26

Martin-Molinero mentioned this pull request Oct 4, 2024

Improve performance of OptionChains DF #8359

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`QCAlgorithm`'s `OptionChain()` api refactor #8334

`QCAlgorithm`'s `OptionChain()` api refactor #8334

jhonabreul commented Sep 19, 2024 •

edited

Loading

QCAlgorithm's OptionChain() api refactor #8334

QCAlgorithm's OptionChain() api refactor #8334

Conversation

jhonabreul commented Sep 19, 2024 • edited Loading

Description

Data frame before:

Data frame after (single option chain):

Data frame after (multiple option chains):

Related Issue

Motivation and Context

Requires Documentation Change

How Has This Been Tested?

Types of changes

Checklist:

`QCAlgorithm`'s `OptionChain()` api refactor #8334

`QCAlgorithm`'s `OptionChain()` api refactor #8334

jhonabreul commented Sep 19, 2024 •

edited

Loading